Risk Minimization and Minimum Description for Linear Discriminant Functions

Statistical learning theory provides a formal criterion for learning a concept from examples. This theory addresses directly the tradeoff in empirical fit and generalization. In practice, this leads to the structural risk minimization principle where one minimizes a bound on the overall risk functional. For learning linear discriminant functions, this bound is impacted by the minimum of two terms – the dimension and the inverse of the margin. A popular and powerful learning mechanism, support vector machines, focuses on maximizing the margin. We look at methods that focus on minimizing the dimensionality, which, coincidentally, fulfills another useful criterion-the minimum description length principle.